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ABSTRACT 

This report discusses the evaluation of the academic 
English language program at the International University of Japan 
(lUJ) , focusing on some common evaluation problems and approaches 
taken at lUJ to solving them and drawing on recent theory. The first 
of these techniques is a systematic approach to program data 
collection using varied kinds of information: qualitative and 
quantitative, formal and informal, and information that is both 
directly and indirectly relevant. Sources of information include test 
scores, student feedback, language teachers, program and university 
records, academic faculty, administrators, alumni, sponsors, and 
other programs. The second technique is a dual focus on evaluation 
process and product. Efforts are made to maximize the time, 
expertise, and motivation of participants and to produce an annual 
program report useful to both program staff and outsiders. The third 
technique involves viewing the program from two additional 
perspectives: (1) having a role as part of a larger "ecosystem," and 
(2) making decisions by balancing each option with the direct or 
indirect costs of an alternative. These techniques have been found 
useful at lUJ for reducing the problems inherent in program 
eval ua t i on . (MSE) 
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EAP PROGRAM EVALUATION IN AN ASIAN CONTEXT: 
A CASE FROM JAPAN 
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INTRODUCTION 

For numerous reasons, language program evaluation is seldom a smooth, 
straightforward process. To begin with, program administrators and curriculum developers 
often have only limited expertise on testing and evaluation. If they do possess the 
appropriate knowledge, it is still no easy matter to win the cooperation of all parties 
involved in order to effectively implement evaluation procedures; and even if all concerned 
arc favorably disposed toward the endeavor, evaluations normally take place at the end of a 
program, when the admmistrators, teachers, and students need to redirect their thinVing to 
the next term or program or project, or perhaps to a vacation. At the International 
University of Japan (lUJ), the English Language Pro-am (ELP) is typical of language 
programs in general in that its evaluation process is subject to each of the above problems, 
but we have also had a certain amount of success in alleviating each of them. 

The purpose of this paper is to describe certain aspects of the lUJ-ELP's evaluation 
process, in the hope that other language program evaluators can either benefit from our 
experiences or can provide us with some better ideas. My description will focus on ways we 
have tried to incorporate some very useful recent work that has been done on language 
program evaluation. I will start by showing how we have applied J,D. Brown's (1989) 
"systematic approach to curriculum improvement and maintenance," and then discuss our 
evaluation process in relation to Michael Long's (1984) ideas on "process and product" in 
evaluation. Fmally, I will show how fruitful ideas: for program evaluation can be generated 
by looking at the program from certain additional points of view: one is is as part of an 
"ecosystem" (HoUiday and Cooke 1982), and the other in terms of "opportunity costs" 
(Swales 1989). 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



V 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



us DEPARTMENT OF EDUCATION 
OHice o' Education*! Research and Improvefnent 

EDUCATIONAL RESOURCES INFORMATJON 
CENTER (ERlCi 

(o^his docufneni has been reproduced as 
received frofT> the person or organization 

originating tt 

Minor Changes have been made to improve 
'eoroduc'ion quality 

• Po<ntS O' v<ew or opintOnS Staled m this dOCu 

ment do not necessaniy represent official 
OPRi posj-i-c^ 0' policy 



PROGRAM BACKGROUND 

The lUJ- ELP was one of the first in a rapidly growing number of English for 
Academic Purposes (EAP) programs m Japan charged with preparing Japanese students 
to do English-medium academic studies. Whereas many of the other programs, such as the 
Japan programs of Temple University and Southern Illinois University, prepare students 
to do the main part of thcii studies in an Er^lish-speaking country, the entire curriculum at 
lUJ Is undertaken on its campus in Niigata Prefecture. lUJ began with one academic 
program, a course in International Relations leading to an M.A. degree; in 1988 it 
established a second graduate school offering an M,BA. in International Management. 
There arc two main student populations: Japanese company employees who are sponsored 
by their employers to study for two years, and students from abroad who are attending on 
j scholarship. These Japanese and non-Japanese populations are about equal in size 

^ (currently about 120 each), and the international students come from over thirty countries. 

« In the ELP, we deal primarily with the Japanese students, because the international students 

\^ tend to be stronger in English and in university-level study skills. All applicants to lUJ take 

TOEFL as part ot the admissions procedure, the ciurcnt range of our students being 450- 
670. The imivcrsity is now in the process of establishing short-term non-degree courses, 
and there is talk of a third graduate school, dealing with International Development, 
pj The ELP started out in 1983 with a full-time staff of one and four temporary 

instructors to run the first Intensive English Program (TEP) before the university formally 
opened. Now, seven years later, we have a full-time staff of seven faculty members, and an 
\ additional five to ten temporary instructors for our summer Intensive English Program, 

r*"^ which has gradually expanded over the years to its current length of t\^ : c weeks. In 

VV addition to the summer lEP, wc offer two terms of aeditcd EAP courses as a cx)ntinuatiT)n 
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of the work wc began in the summer, but we consider the lEP to be the core of our 
curriculum and the time when students can clearly make progress. The lEP mvolves over 
seven hours of class per day, with a daily average of four hours of homework, and a full 
schedule of extracurricular activities. The EEP is currently divided into seven components, 
as shown in Figure 1. 



Figure 1 
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Although we have always been eager to gather information to help in developing the 
curriculum, applying our evaluation findings has always been an ambiguous undertaking, as 
there has never been a clear consensus on the goals of the university curriculum, and the 
addition of new programs has further confused the issue. Therefore, it is somewhat difficult 
fortheELP to judge clearly how successful it has been within its institutional cont^ a.. This 
situation is further complicated by the fact that student success in dealing with the 
university's curriculum may not be closely related to success in terms of what their sponsors 
ejqpect from them after graduation. Although the lUJ situation may be unique in some 
respects, there are ambiguities mvolved in the relationship between any language program 
and its wider institutional and societal context, and a sound evaluation process can go a long 
way toward resolwng them. 



THE "SYSTEMATIC APPROACH 

J. D. Brown defines evaluation as "the systematic collection and analysis of all 
relevant information necessary to promote the improvement of a curriculum and assess its 
effectiveness and efficiency, as well as participants' attitudes within the context of the 
particular institutions involved" (Brown, 1989:223). This definition may seem a bit awkward 
in its length, but the awkwardness itself serves to demonstrate that the concept of 
evaluation is sufficiently complex as to be very difficult to capture in one sentence. I would 
like to focus on two elements in the definition - systematidty and all relevant information - 
and show how they relate to the process of evaluation at lUJ. 

Systematicity. Brown's use of "systematic" can be usefully interpreted in two ways. The 
first interpretation of "systematic" is that the data collection and analysis itself should be 
done systematically. This may seem rather obvious, but it is worth highlighting because this 
sense of "systematic" should not be taken to mean that data which is collected nou- 
systematically can not be part of evaluation, but rather that the evaluators should work 
towird imposing as much systematidty as possible on data, some of which may appear 
very unsystematic at first. For example, comments made in the school cafeteria about the 
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program by academic professors should not necessarUy be disregarded, but if they aix 
deemed to constitute useful information, then efforts should be made to gather a 
representative sample of them and record them consistently. 

The second interpretation of "systematic" is that evaluation functions in a specific way 
as part of a system. A diagram of Brown's proposed model of evaluation wll clarify this 
sense of "systematic" 

Syttematic approach for dMigni&g and mmintaining language curriculum 
(Brown, 1989:236) 
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As can be clearly seen, the system for designing curriculum goes through five stages from 
needs analysis to teaching, with evaluation providing input at each stage, and then the actual 
teaching provides input to revise the previous needs analysis. The most important thing to 
note about this model is that evaluation has a role at every stage, and is thus a constant 
process. It also should be understood that whereas curriculum design can foUow a logical 
linear sequence from needs analysis to teaching, it need not, as evaluation occurring at any 
stage can h^ve an influence on any other stage. This feature in the model g^ves recognition 
to the fact that evaluation is rarely a straightforward process. 

Ail relevant information. The second important element of Brown's definition of evaluation 
to be focused on is the idea of using "all relevant information." Brown lists 24 data- 
gathering procedures which can be grouped into six categories, which can in turn be 
combined into one of two larger categories, according to the criterion of whether the 
evaluator is an "outsider looking in," or a "facilitator drawing information out" (Brovm, 
1989:233). This is a useful way to conceptuaHzc the possibilities for evaluation procedures, 
but to describe the evaluation process at lUJ, it may be more useful to think in terms of 
types of information, and sources of of information. The types of information we use at 
lUJ include qualitativts as wcU as quantitative, informal as well as formal, and information 
which is indirectly as weU as direcdy relevant. The sources of information wc use include 
student test scores, other program and university records, the students themscWcs, the ELP 
faculty, the academic faculties, admimstrators, alumni, sponsors, and other programs. 
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TYPES OF INFORMATION 

Quantitative vs« qualitatiyc* Concerning the use of both quantitative and qualitative 
infonnation, there should not be much controversy: it is well accepted even in pure research 
that not everything can be quantified. However, there are other reasons for balandng out 
quantitative with qualitative data. Among the people who use evaluation reports to make 
decisions, there are doubtless many vAo find numbers most useful and/or impressive, but 
there are certainly others who are more or less innumerate, and there is a third group who 
like to see the presence of numbers to give them a sense of security that the evaluation is 
real, but in fact what they read is the qualitative part. Beretta (1989) cites a number of 
conflicting research findings on this issue. Another reason for not overemphasizing 
quantitative data is that such an emphasis limits the level of quality of the evaluation to the 
evaluator's level of statistical expertise . Of course it can be argued that it is the 
responsibility of a program evaluator be well-versed in research design and staitistics, but 
the greater responsibility is in fact to evaluate the program as effectively as possible given 
the presently available resources. 

Formal vs. infonnal* The use of informaUy gathered information may seem to contradict 
the need for systematidty, but there are good reasons to include it. The first is that people 
are often more candid in informal situations than in formal ones; it's quite possible that 
one can extract better information from some academic faculty members while chatting in 
the hallway than in even the most well-constructed questionnaire. The second reason is 
that the same faculty member is more likely to fill out that questionnaire conscientiously if 
he has recently spoken personaUy with one or more of the people responsible for it. 
Especially in a smaU educational community, and even more espedally in Japan, informal 
contacts can often be considered a necessity before formal requests have any chance of 
being fulfilled. 

Directly vs. indirectly relevant Indirectly relevant information is that which does not seem 
to relate the program as currently conceived, but could play some role in the future. One 
example of the elEfective use of indirectly relevant data at lUJ can be seen during the period 
of the establishment of the School of International Management in affiliation with 
Dartmouth University's business school. At that time, the representatives from the 
Dartmouth side were enthusiastic about incorporating the "Rassias Method" of language 
teaching into the lUJ language curriculum. By researching this method in a timely way (in 
fact it is an audiolingually-oriented method which relies more or less on the teacher's manic 
behavior to keep the students attentive), and including a diplomatic critique of it in our 
program evaluation document, we were able to show them that we were of coiu-se familiar 
with the method, and that we were already incorporating its strong points into oiu- program 
while avoiding its shortcomings. By doing this, we gained their support without having to 
explicitly accept or reject the method they were promoting. 



SOURCES OF INFORMATION 



Test scores* Student test scores are for us, as for most programs, the most compelling 
measure of success. We* have been working hard to develop valid and reliable criterion- 
referenced tests to judge the effect of each component of our program, but we still have a 
long way to go. Moreover, even when we have achievement measures we are satisfied with, 
we will still be left with the equally di£5cult job of explaining them to con-spcdalists outside 
of the program whose decisions and attitudes may affect us. 

Students* Evaluations of our courses by the students, although by no means an accurate 
measure of success, have nevertheless been by far the most useful source of feedback for 
us. We have found that once students realize that their (pinion really is expected, and that 
it will be paid attention to, they often ejqnress their views quite thou^tfully and frankly. A 
lot of fine-tuning of our program has been possible due to student evaluations. One 
example is the level of btensity of our lEP. As could be seen in Figure 1, the program is 
very intensive, and this level must be maintained for twelve weeks. However^ years of 
feedback from students concerning the intensity of the program have shown us that only a 
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few consider the present level to be a hardship, and years when we lightened up the 
schedule a bit, many students felt somehow cheated. 

ELP instructors . All instructors are asked to write out a brief formal evaluation after each 
course, but especially interesting to us is the feedback we get from the instructors who 
come to teach in our summer Intensive English Program. While it is hard for the 
permanent staff to look objectively at the curriculum whidi they both designed and taught, 
our summer instructors are competent BAP teachers who have to implement an intensive 
curriculum without having had input into its content. Where weaknesses exist, the 
temporary instructors can generally be counted on to notice them. 

Progrmm mcd university records. University records provide us with data concerning 
learner performance outside the program, notably their grades in their academic courses 
and their final thesis grade. Since the students' degree of success in their academic 
programs is most often taken to be the ultimate measure of the success of an EAP 
Program, no amount of program-internal testing and evaluation will be sufHcient. The 
problem with using course grades as a measure of success is that we know very little of the 
process by which professors arrived at those course grades. This has certainly been the case 
at lUJ, ¥^ere course grades and level of English ability, or course grades and success in our 
lEP, have not correlated at all. Since nearly all students graduate with respectable grade 
point averages, we have not worried about this situation too much, but we are nevertheless 
vulnerable to criticism from the academic faculty. 

Academic faculty. To pre-empt possible faculty criticism, it is very useful to find out as 
early as possible how professors view their students' language proficiency. The most 
systematic way to do this is through a questionnaire. This was done once during the first 
year of the program, and is now being done a second time (Uehara 1990). Regardless of the 
quality of the information from the questionnaire, it serves the purpose of showing the 
faculty that we care about our program, and if they do not use this convenient opportunity to 
voice their criticisms in a form where they can be constructive, those faculty members may 
be less likely to express their views later in a less constructive manner. 

Administrators. Administrators play an extremely important role in the evaluation 
process, but in a different way than the parties mentioned so far. From time to time we can 
gather good information from them, but in general the crucial flow of information with 
administrators is in the other direction. The important thing is to present the results of 
evaluation activities to them so that they will see good reason to support our program in the 
future. 

Alumni. Alumni should be an excellent source of data about the effectiveness of the 
program, since they have the perspective to see how well the ELP and the lUJ curriculum 
prepared tiiem for their new duties upon return to their companies. Regrettably, we are 
only now beginning to take on this job systematically, and have sent out a pilot 
questionnaire to 50 of our approximately 500 alumni. In terms of anecdotal data, however, 
wc have had many productive informal conversations with alumni when they have returned 
to the campus for various events, and these conversations have led to a number of 
innovations. 

Sponsors Jt may also be worthwhile to see what the students' corporate sponsors think 
about the job we are doing, but it is difficult to obtain this information in any organized 
way, since we often do not even know what level in the spons'^ring company we should 
address. Informal means of data collection may be useful, however. For example, when 
some of the company sponsors attend entrance and graduation ceremonies, our staff could 
take advantage of those opportunities to casually ask a few carefully chosen questions. 

Other progrcins. Since there are many similarities in goals, methodology, and materials 
among EAP programs around the world, it seems irrational that there is not more 
communication among these programs. We have started to gather documentation from a 
number of programs in Jap^n and the United States, and we would be delighted to share 
materials with additional ones. Starting the year before last, we have also l>ecn sending one 
faculty member per year to teach in the Intensive English Program of the World Maritime 
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Universiiy in Sweden, and one their faculty members will join our program this summer. 
This sort oi staff exchange between programs with similar students and goals has already 
had significant benefits for both programs. 



PROCESS AND PRODUCT EVALUATIONS 

In his article on process and product evaluations, Michael Long focuses on process 
evaluation, defining it as "the systematic observation of classroom behavior with reference 
to the theory of (second) language development which underlies the program being 
evaluated" (Long 1984: 415). He pomts out that if product evaluation, carried out 
carefully with a true e3q>erimental design, and not subject to any threats to mtemal validity, 
shows a program to have arrived at the desired outcomes, we still cannot say that the 
curriculum or methodology caused those outcomes without examining actual classroom 
processes. Long does not discourage the use of product evaluations, but rather argues that 
both types are essential 

More specifically, what Long has in mind with process evaluation involves 
experimental research design with control groups and randomization, periodic video or 
audio recording, transcriptions, and careful examination of selected aspects of teacher and 
student behavior motivated by the SLA research literature, such as error correction, or 
ratio of referential to display questions. Although the rigor inherent in this approach is an 
ideal worthy of striving for, there are limitations in the approach Long advocates when 
applied to real programs, especially EAP programs. 

The first limitation is that Long*s approach does not seem to make a distinction 
between research and evaluation. Whereas research should lead to generalizable 
conclusions, evaluation must lead to specific decisions (Isaac & Michael, 1981). As Daniel 
Stufflebeam, a pioneer in modem educational evaluation, put it, "the purpose of evaluation 
is to improve, not to prove." Beretta (1986) argues along the same lines, claiming that 
Long's emphasis on internal validity shows that he does not recognize evaluation as the 
"applied research" it is. The kind of process evaluation outlined by Long could lead to 
answers to some fundamental questions of curriculum developers and teachers in the long 
run, but is not; likely to provide the relevant answers needed in the short nm. 

A second limitation of Long's ^dew of process evaluation is that it does not take into 
account the fact that EAP represents much more than language development. Therefore, it 
is not feasible to evaluate the program in terms of any one theory. At the very least, we 
need an SLA theory and an academic skill learning theory. 

A third and related limitation is the practical one of being constrained by expertise, 
time, and motivation. The rigorous procedure outlined by Long would require 
considerable amounts of each. Without the full cooperation of a good-sized staff of experts 
who buy into the same theories and and accept the same research priorities, it would be 
difficult to see such an evaluation process through to completion. On the other hand, this 
limitation does not imply that the disciplined process evaluation is not a worthy ideal to 
pursue, but merely shows that our expectations cannot be too high at first, while the fu-st 
two limitations serve mostly to caution that other concerns need to be balanced against 
those that Long emphasizes. 

lUJ application of process evaluation. While we cannot claim to be carrying out process 
evaluations according to Long's criteria, we do take steps to ensure that our curriculum is 
carried out as devised. For our summer lEP, we start by sending out course syllabi and 
teaching materials to temporary instructors before they arrive. Upon arrival, we hold pre- 
scrvice component orientation meetings, and component coordination meetings continue 
on an ongoing basis. The director observes every section of every component at least once. 
Additionally, all instructors keep a cumulative lesson plan notebook on their desks, so that 
instructors of the same (or different) components can compare notes, and the component 
coordinators and program director can sec how well the curriculiun is being followed. 

Midterm student evaluations also provide clues to corroborate that the curriculum is 
really being followed, and that is is worthy of following. An additional feature in our next 
summer program will be student committees for each component. These committees will 
consist of a representative from each section, and will meet once a week to discuss with the 
component coordinator and the program director differences in iheu* experiences. 

o 7 
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PROCESS AND PRODUCT OF EVALUATION 

The c^xperience of the lUJ-ELP in relation to program evaluation can be best 
captured by borrowing the terms process and product, and using them in a completely 
different way from the previous section. The process of evaluation will roughly correspond 
to the formative aspects of evaluation, and the products of evaluation will primarily refer to 
the summative aspects of evaluation. 



PROCESS 

The elements of time, expertise, and motivation have been previously mentioned as 
constr^ts affecting the success of program evaluation. What is needed is an evaluation 
process which can maximize the amounts of these elements available. To increase the 
amoimt of time available, it is necessary for evaluation to become a high priority activity, 
and for as many people as possible to be share the work. To increase the amount of 
expertise available, it is necessary to provide direction and encouragement, and to delimit 
the areas of expertise necessary for each staff member. Finally, to increase motivation, 
everyone involved (especially, but not exclusively, the program staff) needs to understand 
how they themselves can benefit from the evaluation. Of course, it is very easy to see how 
these elements overlap and affect each other. 

Time. In the lUJ-ELP, each faculty member has a responsibility for a certain component 
of the curriculum, a responsibility wUch includes evaluation. Dividing the work up among 
seven faculty members makes the workload bearable, and the director is freed to take 
responsibihty for promoting consistency in the evaluation procedures across components, 
and coherence in the curriculum as a whole. 

Expertise* With regard to expertise, we can generally assume sxiffident knowledge in the 
area of curriculum development when we hire our faculty, but this is certainly not the case 
with regard to the evaluation of that curriculum. We started a campaign two years ago to 
increase knowledge or> topics related to evaluation, with more experienced staff giving 
faculty colloquia on topics such as criterion-referenced testing, behavioral objectives, and 
questionnaire design. Starting this year, we are also delegating stages in Brown^s (1989) 
model of curriculum development to the faculty , i.e. one member will be primarily 
responsible for organizing and serving as a resource person for needs analysis, another for 
objectives, etc. 

Motivation. Motivation for evaluation is generated by giving the staff good reason to 
participate fully in the process. In the lUJ-ELP, everyone realizes that the curriculum they 
developed will get even better as a result of the evaluation process, and they also know that 
the degree to winch they get useful information depends on the amount of care they put into 
designing their evaluations. Likewise, the degree to which their students' progress will be 
evident depends on the quality of the tests the instructors devise. 

Another reason to get involved in the process is the opportunity for professional 
development. At lUJ this is especially easy, because faculty promotion is based on a point 
system that recognizes faculty colloquia and conference presentations. The result is that 
individual faculty members explore some new area which is most often related to 
curriculum or evaluation, provide new knowledge to others through a faculty colloquium, 
get feedback from the faculty, develop it further into a conference presentation, get more 
feedback, and then apply the by then well-developed ideas back to their area of the 
curriculum or evaluation process. The recent papers by Hayes (1990) and Uehara (1990), as 
well as the present paper, are all parts of this process. 

An additional point to made about motivation is that the evaluation process at lUJ is 
essentially non-threatening. One reason this is possible is that our emphasis is always on 
curriculum rather than teacher evaluation. DcbrieBngs after class observations by the 
director center around variations in approaches to implemeuting the curriculum, and 
although instructors may add questions to student evaluation forms spedficalty relating to 
their teaching technique, the program-wide questions focus principally on syllabus and 
materials , Of course, anticipating the results of each set of student course evaluations 
brings some anxiety, but since evaluation is now an integrated part of the program culture, 
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this anxiety becomes a routine aspect of the teaching e]q>erience) and translates into 
incentive to further improve the course. ThelUJ-ELP currentty de-^mphasizcs teacher 
evaluation in order to maintain total commitment of the teaching staff to the evaluation 
process and to the professional development that goes along with it. However, as the 
curricuhun gets more and more finely tuned, and commitment to the process is less of an 
issue, it is easy to foresee a time when teacher evaluation may be highligh ted more. 



PRODUCT 

Although the process of evaluation itself serves the formative purpose within the 
program of providing direction for future improvement, it is usually the case that 
administrators and other groups in dedsion-making positions require some product with 
which to assess the effectiveness and efficiency of the program. In the case of the lUJ-ELP, 
we have been fortunate in having very little direct outside interference, but as we grow and 
demand more resources, members of the Policy Shaping Community, or PSC (see Beretta, 
1990) have begun to take more interest in our use of those resources. 

Following the principle of division of labor and expertise elaborated above, we now 
produce an annual Intensive English Program Final Report, with each component 
coordinator writing the sections relevant to that component The emphasis is on readability, 
in terms of vocabulary (no SLA, or evaluation, jargon), balance between qualitative and 
quantitative data, and adjustable length. The report starts with a one-page very broad 
overview by the director, then one-page component overviews by each of the coordmators, 
then one-page descr^)tions of the criterion-referenced tests used, then simunaries of student 
evaluations, and so on, gradually moving into charts and graphs of student performance and 
finally some of the relevant raw data. Our intent is that PSC members will read as little or 
as mud as they wish, but come away with an overall picture of the nature and success of 
the program, and the level of effort and professionalism that we pL't into it. We do not try 
to hide problems, but unless they are problems that the particular reader can potentially 
assist us in solving, we mention them straightforwardly but briefly, with an intended 
solution immediately following. 

We are also in the process of producing an ongoing program document, again divided 
into many short sections, in which we are compiling general program information, a 
description of the curriculum development process, a rationale for our curriculum, brief 
summaries of student performance and evaluation data for year-by-year comparison, etc 
This report Is intended to be useful to new instructors and to people from other EAP 
programs. It also should prove useful to ourselves, in serving as a reason to step back from 
time to time and look at the program as a whole. 
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THE ECOLOGICAL APPROACH 

It is relatively easy to obtain understanding and forge a consensus among the lUJ- 
ELP staff on most program-internal matters, but on issues involving parties both inside and 
outside the program, this harmony is not always the case. In Japan, as in any foreign 
country, the hosts' ways of dealing with things sometimes do not seem to make sense. In 
these situations, the ecological approach proposed by HoUiday and Cooke (1982) provides 
a very useful metaphor. HoUiday and Cooke see language programs as existing within" a 
milieu of attitudes and e3q)ectations of all the parties involved..we treat this milieu as an 
ecosystem within which we have to work. The novelty of our approach lies in the practical 
implications of this view: the need to accords rights of co-existence to §31 the craipeting but 
interdependent elements of the system, and to work the system, rather than against or 
in spite of it, to the greatest extent possible" (HoUiday and Cooke 1982:126). The goal of 
the program is thus to make the best use of kxal features, both promising and unpromising, 
so that the kmg-term viability of the project or program can be assured. In such a situation 
one crucial function of evaluation is to gather the relevant data on local features and the 
interests of all the parties functioning within the ecosystem. This approach to program 
design is especially appropriate for Japan, where the rule is consensus-type decision* 
making, in which the needs of aU relevant parties are typically weighed into a fiual decision. 
Attempts to impose different modes of decision-making almost invariably result in 
fnistratiofL ^ 
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OPPORTUNITY COST CONSIDERATIONS 



John Swales (1989) discusses the importance in curriculum development of 
understanding the decision-making process that goes mto it. He suggests the usefiihiess of 
applying the economic concept opportunity cost » defined as "real or full costs, taking into 
consideration the deficits created by the forced abandonment of other alternatives'' (Swales 
1989: 82). In other words, every decision involves sacrificing options associated with a 
different course of action. For example, deciding to adopt Textbook B eliminates the 
possibility of enjoying the advantages of your current Textbook A, or other textbooks C, D, 
or £, or no fixed textbook at alL In a sense, focusing on opportunity costs can be seen as 
conservatiang smcc it emphasizes the negative implications of any decision, but in fact it 
equally involves evaluating the costs of not innovating. 

A clear-cut case of opportunity cost considerations influencing an lUJ-BLP decision 
occurred in planning for our 1989 lEP, in which we had to decide whether or not to 
introduce a new program component focusing explicitly on grammatical accuracy. 
Although the (Ntogram director and some of the senior staff felt tliat such a course would be 
pedagogically unsound, some of the newer staff members were totally convinced that this 
type of course was necessary. Furthermore, there was an abundance of anecdotal evidence 
that most of our students would appreciate such a course. To reject the course meant that 
the staff members suppcNting it would then be less committed to the program as a whole, 
and during a long intensive program, the probability was high that students would somehow 
find out about this course that had been denied them. Thus, recognizing these opportunity 
costs, we initiated the course. As a result, the staff members in favor of it worked hard on 
the curriculum to make it work, students liked it, their post-test scores were encouraging, 
and even the skeptical instructors ^o were asked to teach a section of it saw some value in 
U. 

The above case was one in viMch opportunity cost considerations aided in making a 
wise curriculum decision. These considerations could be even more applicable when 
making decisions concerning the program^s relationship with its institutional environment, 
for example, in deciding whether the program should offer new short-term courses, or 
whether it should become independent of the parent institution. In such situations. Swales 
(1989) argues that the ecological approaches emphasis on understanding how the system 
works is not enough, and that the concept of opportunity cost can provide more specific 
guidance in making the right strategic decisions. 



CONCLUSION 

Although the literature on language program evaluation is still in its incipient stages, 
it has already offered some sound and usable ideas. From J J>. Brown (1989) we have taken 
an overall framework for curriculum development; from Michael Long (1984) we have 
realized the importance of investigating the process that we assume has produced our 
program results, and of working toward greater scientific disdpline in our evaluation efforts; 
fi-om HoSliday and Cooke (1982) we have gained an appreciation of the need to understand 
our role as a part of a complex larger system; and from John Swales (1989) we have received 
an approach to applying the information we have gathered toward making strategic 
decisions. 

Although the lUJ-ELP still suffers to some degree from most of the recurrent 
problems of program design enumerated by Swales (1989:86), we feel that through our 
evaluation process, we have been able <o make each problem much less severe than it would 
be otherwise; we have also been able to determine with increasing accuracy which 
problems we should vigorously continue to try to solve, and which we should simply accept 
as unfavorable "local features," which may prove in the future to have "positive and 
exploitable aspects" (HoUiday and Cooke, 1982:137). 

If there is anything that other programs can learn from us, it is the benefits of making 
the process and products of evaluation an integral part of program culture. When the 
evaluation process becomes second nature to a program staff, any weaknesses in the 
procedures themselves will be overshadowed by the commitment of the staff to the program 
and the virtual guarantee of ongoing improvement in the future. Furthermore, ykhcn the 
regular reporting of evaluation findings becomes an established practice, in a way that 
adequately considers the needs of each audience, the chances of obtaining enhanced 
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^ * * institudooai support are sure to increase. Finally, it seems reasonable to assume that there 
arc many things that EAP programs around the world can learn from each other; sound 
evaluation practices of course contribute to ha^^ something to say, but by far the most 
important first step is simply communication. We at the International University of Japan 
look forward to sharing ideas and e:q>eriences with other EAF programs. 
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